Papers with TRUE benchmark
WeCheck: Strong Factual Consistency Checker via Weakly Supervised Learning (2023.acl-long)
Copied to clipboard
| Challenge: | Existing factual consistency metrics are often uncontrollably generating text that is factually inconsistent with inputs. |
| Approach: | They propose a weakly supervised framework that is directly trained on actual generated samples from language models with weakly annotated labels. |
| Outcome: | The proposed framework improves on the TRUE benchmark by 3.3% over existing methods with 435M parameters. |
With a Little Push, NLI Models can Robustly and Efficiently Predict Faithfulness (2023.acl-short)
Copied to clipboard
| Challenge: | Conditional language models generate unfaithful output that is not supported by their input . this jeopardizes trust in real-world applications, raising a need for automatic faithfulness metrics. |
| Approach: | They propose to augment conditional language models with robust inference procedures to improve faithfulness. |
| Outcome: | The proposed approach outperforms existing models on the TRUE benchmark. |
A synthetic data approach for domain generalization of NLI models (2024.acl-long)
Copied to clipboard
| Challenge: | Natural Language Inference (NLI) datasets are important benchmark tasks for LLMs . however, their realistic performance on out-of-distribution/domain data is less well-understood . a T5-small model trained with our data improves around 7% on average compared to the best alternative dataset . |
| Approach: | They propose a new approach for generating NLI data in diverse domains and lengths . they show that models trained on this data have the best generalization to completely new downstream test settings . |
| Outcome: | The proposed model can be trained on datasets with high-quality examples with meaningful premises and high accuracy. |
TrueTeacher: Learning Factual Consistency Evaluation with Large Language Models (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods for evaluating factual consistency are limited in their effectiveness. |
| Approach: | They propose a method for generating synthetic data by annotating diverse model-generated summaries using a LLM. |
| Outcome: | The proposed method outperforms state-of-the-art models and the LLM teacher on TRUE benchmarks. |
Are LLMs Better than Reported? Detecting Label Errors and Mitigating Their Effect on Model Performance (2025.emnlp-main)
Copied to clipboard
| Challenge: | Recent advances in large language models (LLMs) offer new opportunities to enhance the annotation process, particularly for detecting label errors in existing datasets. |
| Approach: | They propose to use an ensemble of large language models to flag mislabeled examples by using an LLM-as-a-judge approach to detect label errors in existing datasets. |
| Outcome: | The proposed method improves label accuracy and consistency in large language models. |